NAVAL 

POSTGRADUATE 

SCHOOL 


MONTEREY,  CALIFORNIA 

THESIS 


A  TEST  METHODOLOGY  FOR  RELIABILITY 
ASSESSMENT  OF  COLLABORATIVE  TOOLS 

by 

Brenda  Joy  Powers 
September  2004 

Thesis  Advisor:  Mantak  Shing 

Thesis  Co-Advisor:  Neil  Rowe 


Approved  for  public  release;  distribution  is  unlimited 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


REPORT  DOCUMENTATION  PAGE 


Form^££mved^OMB^No^^0704^018^^^^ 
Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time 
for  reviewing  instruction,  searching  existing  data  sources,  gathering  and  maintaining  the  data  needed,  and  completing 
and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this 
collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  headquarters  Services, 
Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202- 
4302^ndtotheOfficeofManagementandBudget^aperworkReductionProiect(0704-0188^^ 

I.  AGENCY  USE  ONLY  (Leave  blank)  2.  REPORT  DATE  3.  REPORT  TYPE  AND  DATES  COVERED 

September  2004  Master’s  Thesis 

4.  TITLE  AND  SUBTITLE:  A  Test  Methodology  for  Reliability  Assessment  5.  FUNDING  NUMBERS 
of  Collaborative  Tools 

6.  AUTHOR(S)  Brenda  Joy  Powers 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES)  8.  PERFORMING  ORGANIZATION 

Naval  Postgraduate  School  REPORT  NUMBER 

Monterey,  CA  93943-5000 

9.  SPONSORING  /MONITORING  AGENCY  NAME(S)  AND  10.  SPONSORING/MONITORING 

ADDRESS(ES)  AGENCY  REPORT  NUMBER 

N/A 

II.  SUPPLEMENTARY  NOTES  The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official 

policy  or  position  of  the  Department  of  Defense  or  the  U.S.  Government. _ 

12a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT  12b.  DISTRIBUTION  CODE 

_A££roved^or_£ublicj|elease^_distribi4ionjs_unlimite^^^^^^^^^^^^^_ 

13.  ABSTRACT  (maximum  200  words) 

In  the  past  ten  years,  military  operations,  as  now  evident  in  Iraq,  involve  both  joint-  allied  and  coalition 
forces.  The  evolving  joint-  and  coalition-warfare  environment  presents  coordination  challenges. 
Collaborative  tools  can  ease  the  difficulties  in  meeting  these  challenges  by  enabling  highly  interactive  work 
to  be  performed  by  individuals  not  necessarily  geographically  co-located.  Collaborative  tools  will 
revolutionize  the  manner  in  which  distributed  warfighters  interact  and  inform  each  other  of  the  mission¬ 
planning  progress  and  situation  assessment.  These  systems  allow  warfighters  to  integrate  tactical 
information  with  key  combat-support  logistics  data  in  both  joint-  and  coalition-warfare  environments. 
Countless  collaboration  tools  and  knowledge  management  systems  exist  today.  Unfortunately,  industry  has 
developed  these  tools  and  systems  for  use  primarily  in  exclusive  communities  of  interests,  services  or 
agencies.  The  end  result  is  a  proliferation  of  tools  that  have  not  been  designed  to  operate  under  all  network 
conditions.  Since  network  conditions  are  not  standardized  in  the  joint-  and  coalition-warfare  environment,  it 
is  necessary  to  determine  if  a  collaborative  tool  can  perform  under  limited-bandwidth  and  latency  conditions. 
Currently,  there  are  neither  evaluation  criteria  nor  methodologies  for  evaluating  collaborative  tools  with 
respect  to  performance  reliability.  This  thesis  proposes  a  test  methodology  for  evaluation  of  performance 
reliability  of  collaborative  tools,  and  demonstrates  the  effectiveness  of  the  methodology  with  a  case  study  of 
the  performance  evaluation  of  the  Info  Workspace  collaborative  tool. _ 

14.  SUBJECT  TERMS  Collaboration,  Collaborative  Tools,  Software  Reliability,  Performance,  15.  NUMBER  OF 

Bandwidth,  Latency,  Network,  DISA,  CMC,  DIA,  JIVA,  USJFCOM,  CIE,  Communications  PAGES 
Resiliency,  Test  Case,  IWS,  DCTS,  DoD,  COTS,  MCU,  NGCS,XML,  GROOVE,  WEBBE,  _ 69 _ 


GDSS,  Groupware 

16.  PRICE  CODE 

17.  SECURITY 

18.  SECURITY 

19.  SECURITY 

20.  LIMITATION  OF 

CLASSIFICATION  OF 

CLASSIFICATION  OF  THIS 

CLASSIFICATION  OF 

ABSTRACT 

REPORT 

PAGE 

ABSTRACT 

Unclassified 

Unclassified 

Unclassified 

UL 

NSN  7540-01-280-5500 

Standard  Form  298  (Rev.  2-89) 
Prescribed  by  ANSI  Std.  239-18 

I 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


Approved  for  public  release;  distribution  is  unlimited 


A  TEST  METHODOLOGY  FOR  RELIABILITY  ASSESSMENT  OF 
COLLABORATIVE  TOOLS 

Brenda  Joy  Powers 

B.A.,  Point  Loma  Nazarene  University,  1984 


Submitted  in  partial  fulfillment  of  the 
requirements  for  the  degree  of 


MASTER  OF  SCIENCE  IN  SOFTWARE  ENGINEERING 


from  the 


NAVAL  POSTGRADUATE  SCHOOL 
September  2004 


Author:  Brenda  Joy  Powers 

Approved  by:  Mantak  Shing 

Thesis  Advisor 


Neil  Rowe 
Thesis  Co-Advisor 


Peter  Denning 

Chairman,  Department  of  Computer  Science 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


IV 


ABSTRACT 


In  the  past  ten  years,  military  operations,  as  now  evident  in  Iraq,  involve 
both  joint-Allied  and  coalition  forces.  The  evolving  joint-  and  coalition-warfare 
environment  presents  coordination  challenges.  Collaborative  tools  can  ease  the 
difficulties  in  meeting  these  challenges  by  enabling  highly  interactive  work  to  be 
performed  by  individuals  not  necessarily  geographically  co-located.  Collaborative 
tools  will  revolutionize  the  manner  in  which  distributed  warfighters  interact  and 
inform  each  other  of  the  mission-planning  progress  and  situation  assessment. 
These  systems  allow  warfighters  to  integrate  tactical  information  with  key 
combat-support  logistics  data  in  both  joint-  and  coalition-warfare  environments. 

Countless  collaboration  tools  and  knowledge  management  systems  exist 
today.  Unfortunately,  industry  has  developed  these  tools  and  systems  for  use 
primarily  in  exclusive  communities  of  interests,  services  or  agencies.  The  end 
result  is  a  proliferation  of  tools  that  have  not  been  designed  to  operate  under  all 
network  conditions.  Since  network  conditions  are  not  standardized  in  the  joint- 
and  coalition-warfare  environment,  it  is  necessary  to  determine  if  a  collaborative 
tool  can  perform  under  limited-bandwidth  and  latency  conditions.  Currently, 
there  are  neither  evaluation  criteria  nor  methodologies  for  evaluating 
collaborative  tools  with  respect  to  performance  reliability.  This  thesis  proposes  a 
test  methodology  for  evaluation  of  performance  reliability  of  collaborative  tools, 
and  demonstrates  the  effectiveness  of  the  methodology  with  a  case  study  of  the 
performance  evaluation  of  the  InfoWorkSpace  collaborative  tool. 
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I.  INTRODUCTION 


A.  BACKGROUND 

With  the  increase  of  Joint  Force  and  Coalition  participation  in  recent 
operations  and  exercises,  the  military  is  seeking  a  way  to  support  a  new  way  of 
doing  business.  The  ability  of  geographically  dispersed  team  members  to 
communicate  and  collaborate  has  become  of  paramount  interest.  Collaborative 
software  systems  also  referred  to  as  collaborative  tools,  which  offer  capabilities 
such  as  chat,  video  conferencing,  document  sharing  and  audio  conferencing,  are 
being  looked  to  as  the  solution  to  support  this  new  business  paradigm. 
Numerous  collaborative  tools  are  available  and  they  vary  significantly  in  the 
capabilities  provided.  This  makes  it  a  challenging  task  to  select  the  collaborative 
tool  that  meets  the  performance  requirements  of  all  organizations  involved.  For 
example,  network  environments  are  not  standardized  across  the  Joint  or 
Coalition  Communities.  In  fact,  U.S.  troops  face  a  bandwidth  shortage  that 
dictates  where  ships  are  sent,  when  drones  can  fly,  and  what  kind  of  messages 
sailors  and  soldiers  can  receive.  In  military-planning  operations,  where 
exchange  and  coordination  of  information  between  dispersed  team  members  is 
necessary,  collaborative  tools  must  produce  results  within  acceptable  time 
intervals. 

Thus,  evaluating  the  performance  reliability  of  a  collaborative  tool  is  more 
critical  than  ever,  as  is  determining  if  a  collaborative  tool  can  perform  under 
limited-bandwidth  and  latency  conditions.  Currently,  there  are  not  any  standard 
evaluation  criteria  or  test  methodologies  for  evaluating  the  performance  of 
collaborative  tools. 

This  thesis  presents  the  author’s  test  methodology  for  the  evaluation  of 
performance  reliability  of  collaborative  tools.  This  thesis  includes  application  of 
the  test  methodology  during  the  test  and  evaluation  of  IWS  performance 
characteristics  and  reports  the  results. 
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B.  THESIS  ORGANIZATION 

This  chapter  gave  a  brief  introduction  to  the  problem  and  the  motivation  of 
the  research.  Chapter  II  presents  an  overview  of  collaboration,  collaborative 
tools,  and  their  use  in  military  operations.  In  Chapter  III,  four  collaborative  tools 
currently  used  in  military  operations  are  discussed.  Chapter  IV  presents  a 
detailed  description  of  the  author’s  test  methodology.  Chapter  V  documents  how 
the  test  methodology  was  used  to  evaluate  the  performance  of  a  particular 
collaborative  tool  and  presents  the  results  collected.  Chapter  VI  contains  a 
conclusion  and  recommendations  on  future  work. 
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II.  THE  ART  OF  COLLABORATION 


A.  OVERVIEW 

Human  collaboration  has  been  a  subject  of  philosophy  and  social  science 
throughout  History  [JWOD86].  Many  disciplines,  such  as  sociology  and 
anthropology  are  concerned  with  how  people  live  and  work  together. 
Collaboration  is  defined  by  Webster  as,  “The  act  of  working  together  united  in 
labor”.  [WEB1913] 

Collaboration  is  the  basis  for  bringing  together  the  knowledge,  experience 
and  skills  of  multiple  team  members  to  the  development  of  a  new  product  or  plan, 
in  a  way  that  is  more  effective  than  individual  team  members  performing  narrow 
tasks  individually.  Gutwin  and  Greenberg  [GGOO]  have  defined  the  “mechanics 
of  collaboration”  as  the  basic  operations  that  must  be  accomplished  to  achieve  a 
shared  task.  Another  view  [PG03]  identifies  four  categories  of  mechanics:  explicit 
communication  and  information  gathering  (for  communication)  and  management 
of  shared  access  and  transfer  (for  coordination).  Explicit  communication  is 
fundamental  to  collaboration  and  involves  three  types:  spoken,  written  and 
gestural.  The  other  category  for  communicating  is  information  gathering,  which 
can  involve  five  types  of  information:  basic  group  awareness  (who  is  working  and 
what  they  are  working  on);  activity  information  from  objects  (seeing  the  impact  of 
manipulation  on  objects);  activity  information  from  people’s  bodies  (watching  how 
people  act);  visual  evidence;  and  overhearing  others’  explicit  communications. 
Management  of  shared  access  concerns  the  manner  in  which  objects  are 
accessed  and  used.  There  are  three  activities  that  must  be  considered  with 
respect  to  this:  obtaining  a  resource,  reserving  a  resource  for  future  use,  and 
protecting  someone’s  work.  Coordination  may  involve  the  transfer  of  objects  and 
tools  between  individuals  to  ensure  that  a  task  is  divided.  Two  activities  are 
associated  with  this:  handoff  (when  an  object  or  tool  is  transferred)  and  deposit 
(when  a  resource  is  put  in  a  particular  place  to  retrieve  it  later). 
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The  following  sections  discuss  two  other  important  aspects  of 
collaboration. 

1.  Resources 

A  collaborative  process  requires  adequate  resources  so  that  team 
members  have  time  to  effectively  collaborate.  For  example,  too  often  an  engineer 
or  warfighter  is  assigned  to  a  team,  but  is  so  busy  fighting  fires  or  responding  to 
crises  that  he  or  she  does  not  have  the  time  to  effectively  collaborate. 
Management  must  provide  adequate  time  and  money  to  support  collaboration. 

2.  Teamwork 

Effective  teamwork  is  required  for  collaboration.  Team  members  must 
trust  and  respect  one  another.  There  must  be  a  willingness  to  accept  input  from 
others  and  open  communication.  There  are  often  conflicting  goals,  so  decision¬ 
making  must  be  collaborative.  This  is  shown  in  Figure  1.  [KC02] 


Assertiveness 


Competition 

- ► 

Collaboration 

(I  lAiin.you  lose) 

(Win-iAiin) 

Avoidance 

(I  don't  care) 


Accommodation 

(You  lAiin,  I  lose) 


Cooperativeness 


Figure  1:  Collaboration  Model 


This  Figure  shows  two  axes.  Cooperativeness  and  Assertiveness.  A  low 
degree  of  assertiveness  and  cooperativeness  represents  avoidance  of  an  issue 
or  the  approach  of  "I  don’t  care".  A  high  degree  of  cooperativeness  and  a  low 
degree  of  assertion  represents  accommodation;  a  high  degree  of  assertiveness 
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and  a  low  degree  of  cooperativeness  represent  competition.  Compromise,  the 
approach  of  "Sometimes  I  win  and  sometimes  I  lose",  represents  a  moderate 
degree  of  both  assertiveness  and  cooperativeness;  a  high  degree  of  cooperation 
and  assertiveness  represents  the  basis  for  a  "win-win"  collaborative  approach. 
The  key  to  the  latter  is  to  creatively  search  for  solutions  that  can  mutually  satisfy 
the  needs  of  the  team  rather  than  focusing  on  competing  solutions  that  involve 
tradeoffs  or  are  mutually  exclusive. 


B.  COLLABORATIVE  SOFTWARE  SYSTEMS 

Collaborative  software  systems  or  tools  facilitate  the  sharing  of  information 
and  resources  among  individuals  across  geographic  and  temporal  boundaries. 
These  tools  should  accommodate  all  variations  of  interpersonal  and  group 
interactions,  including  one-to-one,  one-to-many,  many-to-one  and  many-to-many. 

Ideally,  the  tools  should  be  dynamic  in  nature  and  have  the  flexibility  to 
support  formal,  informal  and  ad  hoc  collaborations.  They  must  also  be  very 
natural  and  intuitive  to  the  user  as  to  where  to  find  collaborators  and  how  to 
interact  with  them.  A  collaborative  tool  should  accommodate  real-life  situations, 
such  as  interruptions,  and  still  ensure  work  can  be  resumed  seamlessly. 

In  collaborative  software  systems,  at  every  stage  of  the  task,  information  is 
being  processed.  These  processes  -  information  gathering,  information  sharing 
and  information  transfer  -  can  be  thought  of  as  a  continuing  cycle  which  involves 
identification  of  information  needs,  information  acquisition,  information 
organization  and  storage,  information  distribution,  and  information  use  [CC03]. 
These  processes  [PG03]  occur  continually  in  a  collaborative  software  system. 

Collaborative  tools  provides  value  for  an  organization  by  enabling 
dispersed  members  of  a  work  group  to  communicate,  share  knowledge,  and 
develop  plans  and  products.  It  has  been  shown  that  collocation  facilitates 
communication  and  collaboration.  Physical  collocation  is  the  best,  but  virtual 
collocation  through  communication  mechanisms  and  collaborative  tools  is  the 
next  best  alternative  with  dispersed  team  members.  Collaboration  can  happen 
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synchronously  where  all  participants  view  information  and/or  meet  at  the  same 
time,  or  asynchronously  where  participants  view  information  and  provide 
feedback  at  different  points  in  time.  Collaborative  tools  become  increasingly 
valuable  as  more  people  use  it.  For  example,  calendaring  becomes  more  useful 
when  more  people  are  connected  to  the  same  electronic  calendar  and  choose  to 
keep  their  individual  calendars  up-to-date.  [WIQ] 

The  selection  and  use  of  collaborative  tools  will  be  based  on  technology 
availability  and  cost;  geographic  dispersion  and  related  time  zone  differences; 
need  for  access  by  partners,  suppliers  and  customers;  product  complexity  and 
degree  of  technical  issues;  and  other  factors. 

There  are  three  different  categories  of  collaborative  tools: 

1.  Electronic  communication  tools  send  messages,  files,  data,  or 
documents  between  people  and  hence  facilitate  the  sharing  of 
information.  Some  examples  are  e-mail,  faxing,  voice  mail,  and 
Web  publishing. 

2.  Electronic  conferencing  tools  facilitate  the  sharing  of  information  in 
a  more  interactive  way.  Examples  include  data  conferencing,  voice 
conferencing,  video  and  audio  conferencing,  discussion  forums,  a 
virtual  discussion  platform  to  facilitate  and  manage  online  text 
messages,  chat  rooms,  and  electronic  meeting  systems  (EMS). 

3.  Collaborative  management  tools  facilitate  and  manage  group 

activities.  Examples  include  electronic  calendars  (time 

management  software)  that  schedule  events  and  automatically 
notify  and  remind  group  members,  project-management  systems 
that  schedule,  track,  and  chart  the  steps  in  a  project  as  it  is  being 
completed,  workflow  systems  that  provide  collaborative 

management  of  tasks  and  documents  within  a  business  process, 
and  knowledge-management  systems  that  collect,  organize, 
manage,  and  share  information. 
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C.  MILITARY  USE  OF  COLLABORATIVE  TECHNOLOGY 

Collaboration  has  become  a  buzzword  in  the  military  community  these 
days.  The  promise  is  enticing,  getting  warfighters  working  together  online  to 
solve  problems  faster  and  more  responsively  to  customer  needs.  It  is  also  hoped 
to  perhaps  save  on  travel  and  communications  costs  in  the  process. 

Migration  to  collaborative  tools  is  inevitable,  as  in  the  near  future, 
warfighters  will  be  distributed  among  land-based  (physical  or  geo-located) 
command  centers,  mobile  (e.g.,  ship/submarine-board  or  vehicle-based) 
command  centers,  and,  more  importantly,  virtual  command  centers  which  have 
no  physical  counterpart.  How  these  distributed  warfighters  interact  and  inform 
each  other  of  mission  planning  progress  and  situation  assessment,  and  how  they 
establish  their  battle  rhythm  [JDPO02],  is  of  utmost  importance.  Collaborative 
tools  could  significantly  improve  the  effectiveness  of  personnel  distributed  among 
physical  and  virtual  command  centers. 


7 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


8 


III.  SURVEY  OF  COLLABORATIVE  TOOLS 


This  section  provides  an  overview  of  some  collaborative  tools  and  how 
they  are  used  in  military  operations  today. 

A.  INFORMATION  WORKSPACE  (IWS) 

1.  Overview 

InfoWorkSpace  (IWS)  is  a  commercial-off-the-shelf  (COTS)  client-server 
software  solution  that  is  comprised  of  a  suite  of  collaborative  tools  that  facilitate 
enterprise  communication,  data  access,  and  knowledge  management  [IWS04]. 
IWS  is  comprised  of  several  third-party  products  including  Placeware/Microsoft, 
Oracle,  IPIanet/Sun  One  Directory  Server  and  Web  Server,  and  the  Tomcat 
Servlet  Engine/Apache. 

IWS  provides  a  secure  virtual  office  organized  into  buildings,  floors  and 
rooms  where  users  can  build  online  meeting  places  to  interact  on  projects  in  real¬ 
time.  Accessed  via  a  Web  browser  or  Java  client,  it  includes  a  number  of 
features,  including  an  instant-messaging  client  (LaunchPad),  text  chat  (public 
and  private),  audio,  Web  video,  application  casting,  desktop  conferencing.  Virtual 
File  Cabinet,  a  bulletin  board.  Collaborative  Whiteboard  and  shared  Text  Tool, 
threaded  discussions  (news  groups),  mail,  and  a  calendar. 

IWS  can  federate  servers,  which  allows  remote-client  login  to  another 
server  via  authentication  through  the  client’s  home  server.  User  profiles  (such  as 
briefcase  location)  are  stored  on  their  home  servers,  and  file  cabinets  are  stored 
on  their  respective  servers.  Batch  synchronization  between  federated  servers  is 
performed  whenever  a  server  comes  online  or  joins  the  federation.  Incremental 
synchronization  is  performed  in  real  time  as  an  account  is  updated.  This 
includes  inputs  from  the  Lightweight  Directory  Access  Protocol  (LDAP)  directory 
server,  user  and  group  administration  panels,  and  user  login.  Data  synchronized 
among  federated  peers  includes  User/Group  ID,  Home  Host  ID,  Name, 
Distinguished  Name  (LDAP),  Account  Last  Modified,  User  Display  Name,  First 


9 


Name,  Last  Name,  Email,  Title,  Password,  Certificates,  Last  Login,  Number  of 
Login  Failures,  Locked  Flag  and  Group  Members. 

2.  Military  Application 

Combatant  commands  deploying  InfoWorkSpace  as  part  of  the 
Collaborative  Information  Environment  (CIE)  include  the  US  Pacific  Command, 
the  US  Southern  Command,  the  US  Central  Command,  and  the  US  European 
Command,  with  other  combatant  commands  scheduled  for  deployment  later  in 
2004.  The  United  States  Joint  Force  Command  (USJFCOM)  also  coordinated  the 
installation  required  for  InfoWorkSpace  to  be  used  in  support  of  Operation  Iraqi 
Freedom. 

Sponsored  by  the  Defense  Intelligence  Agency  (DIA),  the  Joint 
Intelligence  Virtual  Architecture  (JIVA)  touches  not  only  all  direct  U.S. 
intelligence  agencies,  but  also  each  individual  branch  of  the  U.S.  military. 
Through  the  JIVA  program,  Ezenia's  collaborative  software  solution  is 
deployed  worldwide  to  thousands  of  users,  enabling  globally  dispersed 
organizations  within  the  intelligence  community  to  function  more  effectively 
using  InfoWorkSpace.  Since  1998,  JIVA's  usage  within  the  U.S.  Intelligence 
Community  and  the  military's  major  commands  has  enabled  InfoWorkSpace 
to  gain  further  deployments  in  many  branches  of  the  government,  including 
the  U.S.  Air  Force,  Army,  Navy  and  Marine  Corps.  [EZ04] 

B.  DEFENSE  COLLABORATION  TOOL  SUITE  (DCTS) 

1.  Overview 

The  Defense  Collaboration  Tool  Suite  (DCTS)  is  an  integrated  set  of 
applications  providing  interoperable,  synchronous  and  asynchronous 
collaboration  to  U.S.  Department  of  Defense’s  (DoD)  agencies,  combatant 
commands,  and  military  services  [DCTS04].  DCTS  has  voice  and  video 
conferencing,  document  and  application  sharing,  chat  facilities,  whiteboard 
facilities,  and  virtual-workspace  sharing.  It  is  not  a  single  product  but  an  evolving 
set  of  open  standards.  It  has  a  client/server  architecture  comprised  of  client 
workstations  connected  via  a  network  to  centralized  servers.  The  client 
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applications  manage  all  local  data  processing,  user  interface,  and  data  export  to 
other  client-based  commercial-off-the-shelf  (COTS)  software.  DCTS  Version  2.0 
Phase  I  includes: 

•  Microsoft  NetMeeting:  provides  Windows  users  with  multi-point 
data  conferencing,  text  chat,  whiteboard,  and  file  transfer,  as  well 
as  point-to-point  audio  and  video. 

•  Asynchrony  Envoke:  provides  users  of  different  systems  with 
awareness  or  presence  of  other  users,  spaces  and  meetings. 

•  Sun  Microsystems  SunForum:  provides  shared  applications  and 
conferencing  for  PC  and  UNIX  operating  systems. 

•  Digitai  Dash  Server:  provides  space  navigation,  awareness,  shared 
file  space,  access  control,  VTC  conference  joins,  broadcast 
messages  and  system-administration  services. 

•  First  Virtual  Communications  Multipoint  Control  Unit  (MCU): 
provides  multipoint  NetMeeting  sessions. 

•  Microsoft  SQL  Server:  provides  a  database  and  analysis  for 
e-commerce,  line-of-business  products,  and  data  warehousing. 

2.  Military  Application 

The  DCTS  standard  suite  and  several  certified  collaboration  tools  were 
deployed  during  operations  in  Afghanistan  and  Iraq  in  support  of  both  deliberate 
and  crisis  action  planning.  As  of  January  2004,  DCTS  V2  P1  is  installed  at  138 
sites  worldwide,  at  all  combatant  commands,  major  components  and  services, 
with  another  218  planned  for  2004.  DCTS  will  remain  in  place  until  the  Next 
Generation  Collaboration  Service  (NGCS)  is  on-line  in  2005  or  2006. 


C.  GROOVE  WORKSPACE 
1.  Overview 

Groove  is  an  application  to  facilitate  collaboration  and  communication 
among  small  groups  [GRV04].  It  is  a  commercial  product  invented  by  Lotus 
Notes  creator  Ray  Ozzie.  A  Groove  user  creates  a  "workspace"  and  invites  other 
people  into  it.  Each  person  who  responds  to  an  invitation  becomes  a  member  of 
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that  workspace  and  is  sent  a  copy  that  is  installed  on  his  or  her  hard  drive.  When 
any  one  member  makes  a  change  to  their  copy,  that  change  is  sent  to  all  copies 
for  update. 

Groove's  basic  services  (including  security,  messaging,  store-and-forward 
delivery,  firewall  transparency,  ad  hoc  group  formation,  and  change  notification) 
may  be  customized  with  tools.  These  include  a  calendar,  discussion  support,  file 
sharing,  an  outliner,  pictures,  a  notepad,  a  sketchpad,  and  a  Web  browser. 

2.  Military  Application 

Groove's  software  connects  intelligence  agents  working  in  offices  around 
the  world.  "Groove  has  always  met  the  security  requirements  we  deal  with  in  our 
intelligence  work,  and  over  the  years  the  performance  has  become  faster  and 
requires  less  bandwidth,"  says  Kevin  Newmeyer,  program  director  for  the  Inter- 
American  Committee  Against  Terrorism  in  Washington,  D.C. 


D.  WEBBE 

1 .  Overview 

The  WEBBE®  MX  Server  is  designed  for  real-time  messaging  and 
collaboration.  It  provides  a  highly  reliable,  scalable,  and  easy-to-manage 
infrastructure  [WEB04].  Its  n-tier  design  combines  the  reliability  and  scalability  of 
SQL  with  the  accessibility  and  openness  of  Internet  Information  Services  (IIS).  It 
includes  a  real-time  messaging  service  built  on  a  secure  standards-oriented 
"federated"  architecture.  The  architecture  is  a  peer-to-peer  design  that  is 
distributed  and  modular  with  no  single  point  of  failure  (see  Figure  2).  It  provides 
presence  awareness,  intelligent  routing,  and  guaranteed  message  delivery  to 
member  servers  within  a  federation.  A  single  federation  can  support  up  to  256 
member  servers.  It  uses  the  XML  Distributed  Architecture  which  distributes  real¬ 
time  messages  on  the  IP  network  rather  than  centralizing  applications  at  the 
network  core.  All  WEBBE  messages  are  considered  delivered  only  when 
retrieved  and  explicitly  acknowledged  by  the  destination  client;  if  no 
acknowledgement  is  received,  the  server  will  store  the  message  for  later  delivery. 
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Figure  2:  WEBBE  Architecture 
2.  Miiitary  Appiication 

In  June  of  2003  Webbe  successfully  passed  the  DoD  interoperability 
certification  test  for  inclusion  into  DCTS.  [WEB04]  A  GSA  contract  was  awarded 
in  May  2004  to  upgrade  the  Webbe  Instant  Messaging  Tool  Software 
Server/Client  to  support  the  Special  Operations  Mission  Planning  Environment 
(SOMPE)  mission.  Other  commercially  available  instant  messaging  tools  do  not 
provide  the  voice  instant  messaging  and  highly  compressed  audio  needed  for 
this  application. 
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IV. 


RELIABILITY  ASSESSMENT  OF  COLLABORATIVE  TOOLS 


A.  SOFTWARE  RELIABILITY 

IEEE  610.12-1990  defines  reliability  as  “The  ability  of  a  system  or 
component  to  perform  its  required  functions  under  stated  conditions  for  a 
specified  period  of  time.”  [IEEE90].  Classical  reliability  theory  generally  deals 
with  hardware.  But  the  reliability  of  software  should  be  measured  and  evaluated 
as  it  is  for  hardware. 

While  the  author’s  focus  is  on  the  importance  of  measuring  the  reliability  of 
collaborative  tools  in  terms  of  performance,  it  is  worth  noting  that  measuring  the 
usability  of  collaborative  tools  is  the  most  researched.  Usability  addresses  the 
relationship  between  tools  and  their  users.  In  order  for  a  tool  to  be  effective,  it 
must  allow  intended  users  to  accomplish  their  tasks  in  the  best  way  possible. 
Usability  is  one  of  the  focuses  of  the  field  of  human-computer  Interaction.  As  the 
name  suggests,  usability  has  to  do  with  bridging  the  gap  between  people  and 
machines.  The  Design  of  Everyday  Things,  by  Don  Norman  is  a  book  that  talks 
about  the  importance  of  usability  in  design.  It  doesn’t  focus  on  computers,  but 
about  all  kinds  of  other  things  that  we  suffer  from  every  day  (i.e  doors  that  pull 
open  but  they  look  like  they  should  push  open)  that  are  examples  of  bad  user 
interface  design. 


B.  PERFORMANCE:  A  RELIABILITY  ATTRIBUTE 

Performance  can  be  characterized  by  three  criteria:  throughput,  response 
time,  and  utilization.  Throughput  is  the  number  of  tasks  accomplished  per  unit  of 
time,  for  example  the  transactions  per  second.  Response  time  is  the  time 
elapsed  between  input  arrival  and  delivery  of  output.  Utilization  is  the 
percentage  of  time  a  component  is  busy.  [BROO] 

There  are  many  different  models  for  software  quality,  but  in  almost  all 
models  reliability  is  incorporated.  ISO  9126  [1991]  defines  six  quality 
characteristics,  one  of  which  is  reliability. 
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C.  RELATED  RESEARCH 

As  early  as  1988,  people  began  evaluating  technology  for  use  in 
collaboration.  Some  of  the  first  collaborative  tools  evaluated  were  group  decision 
support  systems  (GDSS).  In  their  paper  Kraemer  and  King  [KK88]  review 
several  GDSS  that  have  been  configured  to  meet  the  needs  of  groups  at  work, 
and  conduct  experiments.  They  also  present  an  assessment  of  GDSS 
development  and  use  in  the  United  States,  and  trace  the  evolution  of  GDSS  to 
support  activities  other  than  decision-making,  including  communication  and 
information  processing. 

There  have  been  a  number  of  studies  related  to  evaluating  both 
collaborative  software  and  groupware.  Groupware  is  any  type  of  software 
designed  for  groups  and  for  communication  that  integrates  work  on  a  single 
project  by  several  concurrent  users  at  separated  workstations.  It  was  pioneered 
by  Lotus  Software  with  the  popular  Lotus  Notes  application  running  in  connection 
with  a  Lotus  Domino  server.  Studies  have  been  done  to  determine  how,  and  if, 
inspection  methods  complement  field  methods  for  evaluating  groupware.  In 
particular,  studies  aim  to  discover  what  kind  of  usability  problems  exist,  and 
whether  inspection  methods  can  provide  an  overall  assessment  of  the  usability  of 
a  system.  To  explore  these  issues,  a  user-based  study  of  how  collaborators 
used  the  Teamwave  Workplace  (TW)  groupware  tool  was  performed  and  the 
results  are  reported  in  a  research  paper  [SMGG01]. 

Many  collaborative  software  tools  have  been  developed  in  the  recent 
years  to  accelerate  the  growing  interest  of  many  organizations  to  become 
learning  organizations.  A  very  interesting  paper  on  the  application  of  Bayesian 
Networks  as  an  evaluation  methodology  to  rate  the  suitability  of  a  given 
collaborative  tool  in  supporting  the  mental  model  concept  of  organizational 
learning  has  been  written  to  discuss  this  [ERF02]. 
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D.  TEST  METHODOLOGY  FOR  THE  ASSESSMENT  OF  PERFORMANCE 

RELIABILITY  OF  COLLABORATIVE  TOOLS 

A  number  of  methods  are  used  to  evaluate  the  reliability  and  performance 
of  collaborative  tools.  The  development  of  a  test  methodology  was  complicated 
by  the  lack  of  standards  or  specifications  that  describe  what  collaborative  tools 
should  do  to  assist  in  military  operations.  The  author’s  test  methodology  used  for 
evaluating  the  performance  reliability  of  collaborative  tools  consists  of  the 
following  steps: 

1 .  Define  the  test  objective. 

2.  Identify  the  requirements  of  the  domain  or  mission  for  which  the 
collaborative  tool  will  be  used. 

3.  Identify  performance  objectives  and  define  specific  parameters  for 
assessment. 

4.  Establish  a  test  architecture  (testbed)  based  on  operational 
requirements  and  performance  criteria. 

5.  Document  the  system  configuration. 

6.  Develop  test  cases  for  collaborative-tool  capabilities  based  on 
performance  criteria. 

7.  Identify  relevant  scenarios  for  evaluating  performance  during  test- 

case  execution. 

8.  Execute  tests. 

9.  Report  the  test  results. 
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V.  APPLICATION  OF  TEST  METHODOLOGY:  IWS  CASE 

STUDY 


The  following  sections  provide  an  overview  of  how  the  test  methodology 
was  applied  in  evaluating  the  collaborative  tool,  InfoWorkSpace  (IWS)  version 
2.5.1 .2,  by  Ezenia. 

A.  TEST  OBJECTIVE 

Bandwidth  is  a  critical  issue  for  U.S.  Military  units.  Over-subscribed 
satellite  resources,  line  of  site  radio  limitations,  and  legacy  switching  constraints 
all  contribute  to  severe  bandwidth  limitations.  The  Navy  especially  faces 
challenges  with  connectivity  due  to  bandwidth  limitations  and  occasional  loss  of 
connectivity  with  the  satellites  they  use.  According  to  GlobalSecurity.org  analyst 
Patrick  Garrett,  the  fact  that  every  message  is  transmitted  electronically  -  from 
maintenance  supply  requests  to  food  orders  to  letters  back  home  -  makes  the 
Navy's  network  traffic  jam  even  worse.  [SN03] 

This  research  was  sponsored  by  Defense  Information  Systems  Agency 
(DISA)  Collaboration  Management  Office  (CMO)  to  investigate  the  performance 
characteristics  of  IWS.  These  tests  were  executed  in  the  SPAWAR  SYSTEMS 
CENTER  SAN  DIEGO  Reconfigurable  Land-Based  Test  Site  (SSC  SD  RLBTS) 
lab,  and  examined  the  products’  bandwidth  utilization,  throughput,  and  response 
time  for  each  feature. 

B.  DOMAIN  REQUIREMENTS 

This  research  was  performed  to  determine  to  what  extent  IWS  can  be 
used  when  limited  bandwidth  conditions  exist  in  ship  to  ship  communications. 
The  requirements  identified  in  support  ship-to-ship  communications  are  as 
follows: 

•  The  collaborative  tool  shall  be  able  to  recover  if  communication  is 
broken  -  resiliency. 

•  When  not  in  use,  the  collaborative  tool  must  not  use  excessive 
bandwidth  -  keep  alive  or  feature  sustainment. 
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•  Features  of  the  collaborative  tool  shall  be  able  to  function  under 
real  world  bandwidth  limitations  and  delay/congestion  conditions 
aboard  ships. 

C.  PERFORMANCE  OBJECTIVES 

The  performance  objectives  of  the  testing  were  to  characterize  the 
reliability  of  each  IWS  feature  with  respect  to  communications  resiliency, 
sustainment  capability,  bandwidth  use,  throughput,  and  to  determine  quality 
limitations  (degradation  points)  for  each  capability.  The  parameters  chosen  were 
bandwidth  and  latency  ranges  of  256  kbps  to  10  ms.  These  were  based  on  real 
world  conditions  aboard  ship. 

The  features  we  considered  most  important  to  measure  were  chat,  file 
upload,  whiteboard  and  audio,  since  they  are  used  for  tasks  such  as 
brainstorming  about  a  briefing,  preparing  the  briefing,  analyzing  imagery,  and 
creating  logistical  or  operational  plans. 


D.  TEST  ENVIRONMENT 

In  accordance  with  step  four  of  the  test  methodology,  a  testbed 
architecture  was  established  to  model  the  operational  requirements  for  ship-to- 
ship  collaboration  at  a  reduced  bandwidth.  The  testbed  was  set  up  in  the  RLBTS 
Satellite  lab  located  at  SSC  SD,  Building  606,  Lab  339.  IWS  was  setup  using  a 
simulated  platform-to-platform  network  configuration,  with  two  domains  as  shown 
in  Figure  4.  The  testbed  architecture  included  two  servers  representing  the  two 
platforms  with  three  clients,  each  communicating  via  a  satellite  simulator 
(SATSIM)  that  can  control  bandwidth  and  latency.  The  interaction  between  the 
clients  collaborating  on  various  planning  activities  is  described  in  detail  in 
conjunction  with  the  description  of  the  scenarios  that  were  executed. 
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IWS  Test  Environment 


Figure  3:  Testbed  Architecture 
1.  Hardware  Configuration 

The  hardware  that  was  used  was  that  which  is  required  as  stated  in  the 
IWS  technical  Specifications  Manual.  Current  users  of  IWS  also  use  this 
hardware  configuration.  The  hardware  configuration  and  installed  operating 
systems  for  all  systems  (server  and  client)  used  during  the  performance  tests  are 
documented  in  Table  1. 
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COMPUTER 

NAME 

IP  ADDRESS 

OS 

NIC /SOUND 
CARD 

PROCESSOR/ 

RAM 

HARD 

DRIVE 

IWS1 

192.168.150.22 

Win  2000 

Server 

(SP2) 

Netgear  FAS  10- 
TX 

Dual  Pentium  III 
600 /512MB 

27GB 

(68-pin 

SCSI) 

IWS2 

192.168.169.52 

Win  2000 

Server 

(SP2) 

Netgear  FA310- 
TX 

Dual  Pentium  III 
600 /512MB 

18GB 

(68-pin 

SCSI) 

CLIENT1 

192.168.150.23 

Win  2000 
Pro  (SP2) 

3Com  Etherlink 
XL  (3C905C- 
TX)  /  AC  '97 
Sound  Card 

Pentium  III 
800MHZ/ 

256MB 

27.4GB 

CLIENT2 

192.168.150.24 

Win  2000 
Pro  (SP2) 

3Com  Etherlink 
XL  (3C905-TX)  / 

Pentium  4 
2.0GHZ/ 

512MB 

7.85GB 

CLIENTS 

192.168.169.53 

Win  2000 
Pro  (SP2) 

3Com  Etherlink 
XL  (3C905C- 
TX)/ 

Pentium  III 
800MHZ/ 

256MB 

14.3GB 

CLIENT4 

192.168.169.54 

Win  2000 
Pro  (SP2) 

Netgear  FA310- 
TX 

Pentium  II 
300MHZ/ 

96MB 

4.05  & 
2GB 

CLIENTS 

192.168.150.30 

Win  2000 
Pro  (SP4) 

3Com  3C920 

Pentium  4  1.8 
GHZ /256MB 

37GB 

CLIENTS 

192.168.169.55 

Win  2000 
Pro  (SP3) 

3Com  Etherlink 
XL  (3C905B- 
TX) 

AMD-K6  ™  3D 
processor / 
160MB 

5.68GB 

Table  1:  Hardware  Configuration 

Tests  were  conducted  with  the  SX/12  Data  Channel  Satellite  Simulator 
(SATSIM),  using  the  EIA-530  Interface.  The  maximum  data  rate  is  10 
megabits/second  but  tests  were  conducted  with  a  maximum  data  rate  of  256 
kbps  kilobits/second  for  the  most  part.  Time  delay  of  data  transfer  was 
configured  from  10ms  up  to  1000ms.  A  Bit  Error  Rate  (BER)  of  IE-6  was 
selected.  Test  case  runs  did  vary  the  BER  since  this  would  add  another  variable 
to  the  bandwidth  and  latency  variation,  which  was  the  primary  focus  of  this 
testing. 
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2.  Software  Configuration 

Software  installed  on  the  servers  included  OracleSi  8.1.7,  Jakarta-Tomcat 
3.2.2,  Oracle  Internet  File  System  1.0. 8. 1.0,  AsPerl,  iPlanet  Server  Products  5.0, 
IWS2512  2. 5.1. 2,  IWSSP251  Service  Pack  1,  LiveUpdate  1.7  (Symantec 
Corporation),  Microsoft  Baseline  Analyzer,  Norton  Antivirus  Corporate  Edition, 
PlaceWare  Media  Plug-In,  PlaceWare  Server  on  port  8087,  Windows  2000 
Hotfix,  Windows  2000  Security  Rollup  Package,  Windows  2000  Service  Pack2, 
and  WinZip. 

Software  installed  on  each  client  station  included  Adobe  Reader  6.0, 
InfoWorkSpace  2. 5. 1.2,  Java  2  Runtime  Environment  Standard  Edition 
v1.3.1_07,  LiveUpdate  1.7  (Symantec  Corporation),  Microsoft  Baseline  Security 
Analyzer,  Microsoft  Office  2000  Professional,  Norton  Antivirus  Corporate  Edition, 
QuickCam,  RealPlayer  7  Basic,  WinZip,  ATI  Display  Driver,  InfoWorkSpace 
2. 5. 1.2  Browser  Plugin,  InfoWorkSpace  Webcam,  Internet  Explorer  5.01, 
PlaceWare  Add-In  for  PowerPoint  version  7,  PlaceWare  Media  Plug-In,  and 
PlaceWare  Snapshot  Plug-In. 

The  WAN  Analyzer  software  operates  on  a  Linux  workstation  (a  two- 
Ethernet  port  Rackable  box  was  used)  and  permits  the  monitoring  of  every 
packet  that  passes  either  Ethernet  port  (called  interface  0  and  1  in  the  output  of 
crl_delay).  The  crl_delay  records  the  one-way  latency  of  each  packet  between 
the  two  Ethernet  ports  (or  the  East  and  West  hub  in  our  RLBTS  ShadowLab)  as 
well  as  the  latency  or  "TCP  Round  Trip  Time  (TCPRTT)  for  all  TCP  packets,  and 
outputs  that  in  real-time  or  nearly  so.  At  the  end  of  each  crl_delay  run,  all 
relevant  numbers  (e.g.,  connection  start  time,  connection  duration,  number  of 
packets  during  connection)  and  derived  statistics  of  latencies  are  tabulated  for 
each  open  TCP  connection  during  the  run.  This  tabulation  can  be  referred  to  as 
cri  _delay  "End  Of  Run  Statistics  (EORS)",  from  which  many  useful  conclusions 
about  the  functioning  of  applications  over  the  WAN  can  be  deduced.  Packets  for 
other  protocols  such  as  UDP  (protocol  17)  are  also  recorded. 
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E.  TEST  CASES 

Test  cases  were  developed  for  chat,  file  transfer,  whiteboard,  audio,  video, 
start-up,  feature  sustainment,  resiliency,  cross  server  login  time,  and  combined 
applications  (i.e.  testing  of  multiple  features  simultaneously).  The  test  cases 
focused  on  collecting  data  regarding  performance  of  certain  applications  under 
certain  conditions  including  emulation  of  three  WAN  environments  representing 
high  (256/1 28bps),  medium  (64  kbps),  and  low  (9.6  kbps)  bandwidth.  These 
were  chosen  to  model  real-world  conditions  aboard  ships,  where  at  any  given 
time  each  one  of  these  is  possible  reality.  In  addition,  each  test  included  the 
introduction  of  time  delay  (TD)  in  amounts  from  10ms  to  1000ms  in  some  cases. 


F.  SCENARIOS 

In  accordance  with  step  seven  of  the  test  methodology,  various  scenarios 
were  identified  for  use  in  conjunction  with  the  test  cases.  The  scenarios 
exercised  the  system  features  or  capabilities  in  a  manner  consistent  with  the 
tool's  operational  use.  Table  2  describes  the  scenarios. 


Scenario 

Clients  Involved 

Capability 

Scenariol  -  Point  to  Point 

Clientl/IWSI  talks  to 

remote  Client3/IWS2 

Chat,  Audio 

Scenario2  -  Point  to  Multi- 
Point 

Clientl/IWSI  broadcasts  to 
remote  Client3&4/IWS2 

Chat,  Audio 

Scenarios  -  Multipoint  to 
multipoint 

Clients1,2&5/IWS1  and 

remote  Clients3&4/IWS2  all 
talk  with  one  another 

Chat,  Audio 

Scenario4  -  Cross-server 
login 

Remote  Clients  on  IWS2 
logs  into  IWS1  and  joins  a 
collaboration  session  in 
sidebar  room 

Chat,  file  transfer, 

whiteboard,  video,  audio, 
resiliency,  application 

sustainment 

Table  2:  Test  Case  Scenarios 
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G.  TEST  EXECUTION  AND  PERFORMANCE  RESULTS 

1.  Startup 

Startup  represents  the  product  bandwidth  usage  during  initial  product 
startup.  During  startup,  IWS  downloads  and  installs  data  from  the  server.  The 
client  loads  active  user,  room,  bulletin  board,  and  chat  data,  and  also  downloads 
several  Java  applets  (e.g.  Clock  Service,  text  chat  service,  logging  service. 
Object  ID  Service,  User  preferences  Service,  Room  toolset  administration 
service.  Places  Navigator  Service,  Geospace  Admin  Service,  and  Quality  of 
Service).  The  number  of  bits  transmitted  from  the  client  to  the  server,  over  the 
86  second  time  period  it  took  for  the  client  to  complete  the  startup  process,  was 
approximately  480,024  bits  at  a  bandwidth  of  128kbps  and  latency  of  10ms.  The 
total  number  of  bits  transmitted  from  the  server  to  the  client  over  the  86  second 
time  period  was  868,216.  Thus,  an  average  of  15,677  bits  per  second  was 
exchanged  between  client  and  server  over  the  86  second  initial  startup  period. 
This  information  is  illustrated  in  Figure  4. 

The  overhead  of  the  necessity  of  loading  all  of  this  information  on  initial 
startup  is  the  result  of  the  use  of  a  JAVA  plugin  (JAVA  Runtime  Environment 
1.3.1)  to  launch  the  IWS  browser  based  client  application.  Once  the  initial 
startup  has  been  performed  this  data  is  present  on  the  client. 
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b/w&delay  (kbps/ms)  128/10 


□  Average  Data  Transfer  (bps) ) 


Figure  4:  Startup 

2.  Keep-Alive  (Maintenance  of  Bandwidth) 

Application  sustainment  or  “Keep-alive”  bandwidth  is  product  bandwidth 
usage  during  periods  of  no  end-user  activity.  The  “keep-alive”  values  (i.e.  client 
send,  server  send)  averaged  traffic  from  client  to  server  and  from  server  to  client 
(independent  of  port  number)  over  a  three-minute  period.  This  testing  was 
performed  after  two  clients  had  successfully  logged  into  a  room  on  the  remote 
server.  Test  operators  stopped  all  keyboard  and  user-interface  inputs  and  the 
network  traffic  between  the  local  and  remote  servers  was  recorded  for 
approximately  three  minutes.  The  “keep-alive”  tests  were  executed  for  file 
cabinet  uploads,  whiteboard  use,  and  video  use,  to  determine  if  any 
communication  overhead  was  caused  by  the  feature  having  been  in  use  prior  to 
the  period  of  inactivity.  Test  results  are  provided  in  Figure  5. 
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□  keep-alive 

□  keep-alive  upload 

□  Keep-alive  video 

□  keep-alive  whiteboard 


Figure  5:  Keep  Alive  Bits  per  Second 

Data  analysis  revealed  that  the  four  client  processes  provide  status 
updates  by  opening  an  active  port,  providing  their  data,  and  then  closing  that 
connection. 

An  additional  test  was  performed  to  measure  how  much  network  traffic 
passed  through  the  testbed  wide-area  network  when  two  local  and  two  remote 
clients  where  left  idle  in  SideBarRoomI  on  server  IWS1  for  16  hours  or  57600 
seconds.  Results  showed  that  a  total  of  3,621,831  bytes  passed  through  the 
WAN,  which  is  an  average  of  62.88  bytes/second  or  503.03  bits/second.  This 
provides  an  estimate  of  the  minimum  bandwidth  required  to  maintain  a  session 
between  two  servers  with  four  idle  clients. 

3.  Communications  Resiliency 

Communications  resiliency  was  tested  after  a  collaboration  session  had 
been  successfully  initiated  between  test  platforms.  Test  operators  stopped  all 
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keyboard  and  user-interface  input  between  the  local  and  remote  servers,  and 
then  the  SATSIM  was  brought  down  for  approximately  three  minutes  and  then 
brought  back  up.  The  SATSIM  was  brought  down  by  setting  the  bit  Error  Rate 
(BER)  to  100%. 

In  general,  when  no  applications  had  been  started,  during  SATSIM  down 
time  about  612  kbits  were  transmitted  across  the  WAN  during  4  seconds  while 
attempting  to  restore  the  connection.  Federation  was  eventually  restored; 
however,  the  remote  client  could  not  get  into  IWS1  server  sidebar  rooms. 

When  testing  Chat  Resiliency,  about  75  seconds  after  the  connection  was 
restored,  approximately  9  kbits  were  transferred  across  the  WAN,  but  the  remote 
client  connection  was  rejected. 

During  the  remote  file  transfer  test,  the  local  Clienti  saw  that  the  remote 
Clients  had  left  the  sidebar  room.  The  server  IWS1  indicated  that  the  channel 
was  closed.  The  Clients  file  transfer  progress  bar  stopped  and  a  couple  of 
seconds  later  a  Java  exception  was  thrown.  About  60  seconds  after  the 
connection  was  restored,  approximately  8  kbits  transferred  across  the  WAN. 
About  115  seconds  after  the  connection  was  restored,  approximately  612  kbits 
transferred  across  WAN  during  4.5  seconds  while  the  federation  was  restored 
but  the  remote  client  application  was  locked  up. 

When  testing  Audio  Resiliency,  the  local  Clienti  was  notified  that  the 
remote  Clients  had  left  the  chat  room.  The  local  server  IWS1  reported  that  the 
channel  was  closed.  About  3  minutes  and  40  seconds  after  the  connection  was 
restored,  about  612  kbits  crossed  the  WAN  over  4  seconds  while  the  federation 
was  restored.  When  federation  was  restored.  Clients  was  unable  to  join  IWS1 
sidebar  rooms. 

During  whiteboard  testing,  when  the  SATSIM  was  blocked.  Clients 
received  a  message  box  saying  that  connection  has  been  lost;  when  federation 
was  restored.  Clients  was  unable  to  join  to  IWS1  sidebar  rooms.  Logging  off 
Clients  and  then  logging  back  in  was  the  only  way  Clients  was  able  to  get  the 

objects  restored.  About  5  minutes  and  20  seconds  after  the  connection  was 
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restored,  approximately  612  kbits  transferred  across  the  WAN  during  4  seconds 
while  the  federation  was  restored,  but  the  remote  client  application  was  locked 
up. 

A  general  observation  about  this  group  of  tests  was  that  when  the  servers 
federated  after  the  WAN  connection  was  restored;  it  required  about  610  kbits  of 
data  to  cross  the  WAN,  and  at  9.6  kbps  that  would  take  over  1  minute.  This  can 
be  seen  in  Figure  6. 

During  SATSIM  down  time,  remote  clients  (i.e.  those  on  IWS2  server)  did 
not  detect  they  had  lost  communications.  The  local  server  (IWS1  server)  started 
to  indicate  the  loss  of  remote  participants  approximately  one  minute  after  a 
communications  outage.  This  product  does  have  a  Quality  of  Service  (QoS) 
indicator  display  that  was  informative  for  the  high  bandwidth  scenario,  but  the 
display  indicated  the  same  QoS  in  low  bandwidth  scenario  for  both  SATSIM 
communications  up  and  down.  Upon  re-establishment  of  communications,  the 
remote  clients  received  a  message  from  the  server  indicating  their  attendance  in 
the  meeting  had  been  terminated  and  they  were  not  automatically  reconnected  to 
the  meeting.  Qverall,  the  applications  ceased  working  for  the  remote  client  when 
a  failure  in  communications  occurred. 

File  transfers  between  remote  clients  and  server  that  experienced  a 
communications  outage  during  transfer  necessitated  resending  the  entire  file 
upon  resumption  of  communications.  Note  that  if  remote  clients  attempted  to 
send  data  during  the  SATSIM  down  time,  they  would  receive  an  error  message 
indicating  they  had  lost  their  meeting  connection. 

Resiliency  bandwidth  on  re-federation  test  results  is  provided  in  Figure  6. 
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■  Remote  Private  CHAT 

□  Remote  File  XFer 

□  Remote  File  Whiteboard 

■  Remote  File  Audio 


Figure  6:  Resiliency  Bits  loaded  upon  re-Federation 

4.  Chat 

There  were  four  scenarios  used  in  the  Chat  application  testing  as 
described  in  Table  3.  Each  scenario  was  executed  over  a  90  second  test 
window.  Clients  from  the  local  server  and  remote  server  entered  and  exited 
different  Sidebar  Rooms  and  exchanged  broadcast  messages  with  IWS  clients  in 
that  room,  and  exchanged  Private  messages  with  IWS  clients  in  other  Sidebar 
rooms. 

Chat  testing  was  conducted  at  high,  medium,  and  low  bandwidth  and 
latency  combinations.  The  average  bits  per  second  for  the  data  runs  are  shown 
in  Table  3.  This  was  the  total  bits  transferred  over  the  WAN  divided  by  90.  Chat 
data  flow  was  examined  by  comparing  the  difference  of  having  clients  logged  in 
at  a  local  server  sidebar  room  verses  a  remote  server  sidebar  room.  When  the 
remote  client  was  logged  into  the  sidebar  room,  the  server  to  which  the  room 
belongs  communicated  directly  to  the  remote  client,  not  via  the  remote  clients’ 
server.  This  is  apparently  a  result  of  federation  of  servers. 
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Bandwidth/Delay 

(kbps/ms)1 

256/10 

64/10 

9.6/10 

Scenario  1 

Room  Chat 

11922 

908 

483 

Private  Chat 

1294 

1113 

Room  &  Private 

Chat 

1159 

1873 

Scenario  2 

Room  Chat 

2017 

2161 

1544 

Private  Chat 

2201 

Room  &  Private 

Chat 

2549 

Scenario  3 

Room  Chat 

2698 

4788 

2838 

Private  Chat 

4506 

Room  &  Private 

Chat 

5941 

Scenario  4 

Room  Chat 

10532 

10922 

7429 

Private  Chat 

10678 

Room  &  Private 

Chat 

11379 

Active  User/Send 
note 

10839 

Table  3:  Chat  Test  Cases  and  Results 


Additional  chat  testing  was  conducted  to  determine  the  bandwidth  level 
where  degradation  of  quality  occurred.  Bandwidth  testing  began  at  9.6 
kbps/1 0ms  and  also  checked  4.8/10,  2.4/10  and  1.2  kbps  in  tests.  At  1.2  kbps, 
operators  observed  significant  delays  with  the  chat  from  one  operator  displaying 
at  a  destination  client  about  30  seconds  to  2.5  minutes  after  the  originator  sent 
the  message.  Longer  2.5  min  delays  were  observed  in  scenario  3  when  5 
operators  (and  therefore  more  traffic)  were  active.  Chat  test  results  are  provided 
in  Figure  7. 


1  bandwidth  in  kilobits  per  second  (kbps)  and  delay  in  milliseconds  (ms) 

2  These  values  represent  the  average  bits  per  second  (bps)  transferred  across  the  WAN 
over  the  period  (90  seconds)  of  the  chat  exercise  where  operators  sent  a  text  string  every  10 
seconds  for  a  total  of  10  transmissions. 
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■  Room  Chat 

□  Private  Chat 
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■  Scenario  4 

■  Room  Chat 

□  Private  Chat 

■  Room  &  Private  Chat 

□  Active  User/Send  note 


Figure  7:  Chat  Test  Results 

In  summary,  the  data  showed  that  a  basic  one-client  to  one-client  room 
chat  (scenario  1)  required  approximately  1  kbps  across  the  WAN.  A  one  to  two 
client  chat  (scenario  2)  required  about  double  the  bandwidth  at  2  kbps.  For 
scenario  3,  5  users  required  about  4.5  to  5  kbps.  The  most  intensive  usage  for  a 
single  client  was  when  a  client  logs  in  remotely  to  another  server  sidebar  room 
(scenario  4)  and  this  required  about  10  kbps  for  1  client  to  1  client  chat.  No 
significant  difference  between  the  types  of  chat  was  observed  (i.e.,  room,  private 
or  active  user,  where  active  user  was  measured  when  both  clients  were  logged 
into  the  same  server).  The  apparent  lower  bits  per  second  for  a  9.6k 
communications  path  is  puzzling,  other  than  it  was  verified  that  IWS  does  not 
perform  compression  and  it  may  be  anomalous.  The  results  support 
observations  that  chat  began  to  see  significant  delays  when  the  bandwidth  was 
1.2  kbps  for  scenario  1  and  when  the  bandwidth  was  9.6  kbps  for  scenario  4. 
Additional  delay  time  added  a  proportional  delay  to  observed  response  time  for 
this  application. 
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5.  File  Transfer 

File  transfer  to  both  the  File  Cabinet  and  Briefcase  was  performed  using 
test-procedure  instructions  to  upload  or  download  a  file  from  or  to  any  room’s  File 
Cabinet  on  the  remote  server.  In  addition,  Scenario  4  was  used  such  that  the 
remote  client  file  transfer  across  the  WAN  was  subject  to  the  bandwidth  and 
delay  constraints  listed  in  Table  4.  File  sizes  of  100  kb,  500  kb  and  1  mb  were 
used  during  test  execution.  Test  cases,  comprised  of  file  size  transferred  at  a 
given  bandwidth/delay,  and  the  results  are  shown  in  Table  4. 


b/w& 

delay 

128/ 

250 

64/ 

250 

9.6/ 

250 

128/ 

500 

64/ 

500 

9.6/ 

500 

128/ 

1000 

64/ 

1000 

9.6/ 

1000 

64/ 

10 

9.6/ 

10 

File 

Size 

Average 

bps^ 

100k 

51820 

41, 

105 

7982 

39,077 

33911 

8358 

21884 

21759 

7592 

55892 

8967 

500k 

69714 

58577 

69,245 

45437 

30173 

25134 

61234 

1000 

k 

88906 

59329 

61,779 

48499 

32031 

33822 

64541 

Net  bits4 

100k 

984,580 

1,027, 

625 

993, 

703 

951,54 

6 

983, 

419 

1027, 

904 

962, 

896 

1022, 

664 

1009, 

672 

950, 

168 

1,049, 

184 

500k 

4,601, 

124 

4510, 

429 

4500, 

202 

4498, 

263 

4465, 

604 

4549, 

344 

4531, 

384 

1000 

k 

8,801, 

694 

8840, 

021 

8832, 

114 

8778, 

319 

8776, 

494 

8861, 

376 

8906, 

688 

Time  of 
transfer^ 

100k 

19 

25 

125 

25 

29 

123 

44 

47 

133 

17 

117 

500k 

66 

77 

66 

99 

148 

181 

74 

1000 

k 

96 

149 

143 

181 

274 

262 

138 

Table  4:  File  Transfer  Test  Cases  and  Results 


3  These  values  represent  the  average  bits  per  second  (bps)  transferred  across  the  WAN 
during  the  period  of  the  file  transfer  exercise. 

^  These  values  show  net  bits  transferred  and  provide  an  indication  of  the  accuracy  and 
consistency  of  the  measurements  since  they  should  all  be  about  the  same  across  a  row. 

^  These  values  show  transfer  time  in  seconds  for  the  given  file  size,  b/w  and  delay  condition. 
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A  summary  of  file  transfer  results  for  File  Cabinet  is  provided  in  Figure  8 
where  the  bandwidth  is  held  constant  and  delay  varies,  and  Figure  9  where  the 
delay  is  held  constant  and  the  bandwidth  is  varied. 
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Figure  8:  File  Transfer  Test  Results 

In  summary,  at  a  delay  of  10ms  (low)  the  data  transfer  rate  approaches 
the  limits  of  the  communications  path.  Fligher  delays  in  the  500  to  1000  ms 
range  have  a  significant  impact  on  throughput.  The  larger  the  file,  the  more 
efficient  the  application  appears  at  transfer,  but  this  did  not  make  up  for  overall 
time  to  transfer  a  large  file. 
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Figure  9:  File  Transfer  Test  Results  with  Varying  Bandwidth 

6.  Whiteboard 

Test  execution  began  at  the  bandwidth/latency  rates  of  128/10  ms. 
Scenario  4  was  performed  with  the  test  case  where  all  clients  imported  images. 
Operator  observation  revealed  that  after  the  remote  client6  imported  an  image 
and  it  took  2  seconds  to  render  locally,  it  took  4  seconds  for  the  remote  client  to 
see  it. 

The  same  test  was  conducted  at  the  bandwidth/latency  rates  of  64/1 0ms. 
From  a  user  perspective,  it  appeared  that  after  the  local  Client2  imported  the 
image,  it  took  4  seconds  for  the  remote  Clients  to  see  it. 

The  test  was  then  conducted  at  the  bandwidth/latency  rates  of  9.6/1 0ms, 
and  with  a  slight  variation,  the  addition  of  another  remote  client.  Again,  from  a 
user  perspective,  it  appeared  that  after  the  remote  ClientO  imported  the  image,  it 
took  4  seconds  for  it  to  appear  on  that  client,  30  seconds  for  the  other  remote 
Clients  to  see  it,  and  only  20  seconds  for  the  two  local  clients  1  and  2  to  see  it. 
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Whiteboard  testing  continued  with  Scenario  4  at  the  bandwidth/latency 
rates  of  129/10  ms.  This  test  case  involved  drawing  objects  geometric  shapes 
and  text  on  the  whiteboard.  Results  showed  that  for  the  local  clients  it  took  less 
that  one  second  to  see  the  objects,  and  for  the  remote  client  it  took 
approximately  two  to  three  seconds. 

Tests  conducted  at  9.6/10  ms  showed  a  range  of  results,  depending  on 
the  object  that  was  drawn  or  imported  to  the  whiteboard.  For  example,  when  all 
clients  simultaneously  drew  and  imported  small  images  such  as  circles, 
rectangles,  and  squares  to  the  whiteboard  it  took  five  seconds  for  the  remote 
client  to  see  the  objects  on  its’  whiteboard.  Similarly,  when  a  medium  size  circle 
was  drawn  by  a  remote  client,  it  took  7  seconds  for  the  client  on  the  local  server 
to  see  it.  This  would  seem  to  substantiate  the  author’s  hypothesis  that  the  bigger 
the  object,  the  longer  it  takes  to  render  on  the  whiteboard.  Repeated  tests  did 
confirm  this,  and  larger  objects  took  9-11  seconds  to  render.  During  whiteboard 
testing,  higher  sustained  average  transfers  of  10  to  13  kbps  were  observed  while 
using  larger  objects.  A  summary  of  whiteboard  test  results  is  provided  in  Figure 
10. 
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□  64/10 
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Figure  10:  Whiteboard  Test  Results 

In  summary,  under  optimal  bandwidth  constraints  of  128/10  ms,  when  the 
drawing  action  occurred  from  the  remote  client,  a  3  second  delay  was  noticed 
before  rendering  was  complete.  However,  when  drawing  was  initiated  by  the 
local  client,  the  rendering  was  instantaneous. 

7.  Video 

IWS  uses  WebCam,  a  third  party  application,  for  its  video  capability. 
Video  testing  was  conducted  at  optimal  conditions  (e.g.  256/128/9.6  kbps 
bandwidth  and  10  ms  delay).  The  test  scenario  1  involved  a  local  client  using  the 
Webcam  application  to  display  video  of  a  remote  client  logged  into  the  same 
sidebar  room  session. 

During  IWS  video  testing,  quality  was  poor  and  there  was  a  3  second 
transmission  delay.  The  video  that  was  transmitted  appeared  to  be  still  pictures 
transmitted  every  3  seconds  or  so.  The  picture  appeared  to  update  every  couple 
of  seconds  even  at  higher  bandwidth  of  2Mbps.  Video  of  remote  client  at  the 
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local  client  stopped  updating  during  one  test  run,  so  the  Webcam  application  was 
stopped  and  restarted  to  restore  the  video  application.  At  9.6  kbps,  remote  video 
continued  to  update  but  at  a  slower  rate  of  7-10  seconds.  There  was  difficulty  in 
setting  up  testing,  and  the  solution  was  to  reboot  the  servers  until  we  got  the 
video  application  to  work.  A  summary  of  the  video  test  results  is  provided  in 
Figure  1 1 . 


Figure  1 1 :  Video  Test  Results 

In  summary,  for  the  most  basic  scenario  1  where  one  client  talks  to 
another  client  across  the  WAN  the  approximate  average  bandwidth  was  28  kbps. 
The  picture-update  rate  corresponds  with  data  observations  that  the  video  data 
was  transmitted  approximately  every  2-3  seconds  across  the  WAN. 

8.  Audio 

The  IWS  Audio  or  Voice-over-IP  capability  was  tested  for  each  scenario 
listed  in  Table  2  at  the  bandwidth/latency  rates  of  128/10,  64/10,  32/10,  and 
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9.6/10  kbps/ms.  The  times  were  the  amount  of  time  it  would  take  for  all  clients  to 
receive  the  full  one-way  voice  stream. 

Various  problems  began  to  show  up,  such  as  choppy  and  slightly  delayed 
voice,  and  the  quality  seemed  to  degrade  over  time.  The  test  cases  and  their 
results  are  described  in  this  section. 

During  the  first  test  case,  the  local  client  1  used  the  private  chat/talk 
function  to  transmit  a  single  10  sec  voice  count  from  1  to  10  followed  by  "over",  to 
which  the  remote  client  3  responded  with  the  same  count  followed  by  "over". 
Test  scenario  2  involved  a  local  client  talking  to  2  remote  clients.  Test  scenario  3 
is  defined  as  clienti  talking  to  client  3  and  client  2  talking  to  client  4  (two  separate 
remote  conversations).  When  audio  deterioration  was  observed,  a  paragraph 
was  read  instead  of  the  count  from  1  to  10  followed  by  "over";  this  was  to 
determine  if  the  audio  was  recognizable  since  the  counting  was  easier  for  the 
receiver  to  understand  given  the  receiver  knew  what  to  expect.  Private  and 
broadcast  audio  were  tested  to  see  if  a  different  amount  of  bandwidth  were 
required  for  these  functions.  Test  cases,  at  a  given  bandwidth/delay,  and  the 
results  are  shown  in  Table  5. 


b/w&delay 

128/10 

64/10 

32/10 

9.6/10 

128/250 

64/250 

32/250 

9.6/250 

Scenario  1, 

Average 

bits 

transferred 

18060 

18787 

19088 

8829 

18704 

19051 

17117 

8694 

Net  bits 

transferred 

(bits) 

433448 

319376 

362664 

335504 

392784 

361960 

359456 

330360 

Test 

Duration 

(seconds) 

24 

17 

19 

38 

21 

19 

21 

38 

b/w&delay 

128/500 

64/500 

32/500 

9.6/500 

128/1000 

64/1000 

32/1000 

9.6/1000 

Scenario  1, 

Average 

bits 

transferred 

18424 

18530 

19087 

8944 

18737 

17612 

17733 

8183 

39 


Net  bits 

transferred 

(bits) 

386904 

370592 

362656 

357760 

449680 

352232 

354656 

343672 

Test 

Duration 

(seconds) 

21 

20 

19 

40 

24 

20 

20 
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b/w&delay 

128/10 

64/10 

32/10 

9.6/10 

Scenario  2 

51282 

34264 

Net  bits 

transferred 

(bits) 

512816 

513952 

Test 

Duration 

(seconds) 

10 

15 

Scenario  3 

44570 

43644 

32443 

8594 

Net  bits 

transferred 

(bits) 

802256 

785592 

811080 

429712 

Test 

Duration 

(seconds) 

19 

18 

25 

50 

Scenario  2 

39866 

32926 

Net  bits 

transferred 

(bits) 

398664 

691440 

Test 

Duration 

(seconds) 

10 

21 

b/w&delay 

at 

19.2/10 

at  40/10 

Scenario  3 

21904 

39706 

Net  bits 

transferred 

(bits) 

635224 

794128 

Test 

Duration 

(seconds) 

29 

20 

Table  5:  Audio  Test  Cases  and  Results 


A  summary  of  the  audio  test  results  is  provided  in  Figure  13.  The  values 
in  Figure  12  represent  the  average  bits  per  second  (bps)  transferred  across  the 
WAN  during  the  period  of  the  audio  test  exercise. 
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:□  Scenario  1 
;■  Scenario  2  Private 

□  Scenario  3 

□  Scenario  2  Broadcast 


Figure  12:  Audio  Bits  Test  Results 

In  summary,  for  scenario  1,  where  the  local  client  communicated  to  the 
remote  client  across  the  WAN,  the  bandwidth  was  18-19  kbps.  This  corresponds 
with  observations  that  at  9.6  kbps  the  audio  was  not  functioning  at  all.  Increasing 
the  delay  to  1000ms  seemed  to  slow  the  net  transfer  by  approximately  1  kbps. 
For  scenario  2,  where  one  the  local  client  was  transmitting  voice  to  two  remote 
clients  across  the  WAN,  the  approximate  bandwidth  measured  was  between  33- 
51  kbps.  This  corresponds  with  observations  at  32  kbps  that  the  audio  quality 
had  deteriorated,  and  was  choppy  and  reverberated.  For  scenario  3,  where  2 
separate  audio  sessions  were  established  across  the  WAN,  the  approximate 
bandwidth  measured  was  39-45  kbps.  This  corresponds  with  observations  at  40 
kbps,  when  gaps  in  the  speech  seemed  to  degrade  significantly,  over  the  time  of 
the  audio  exercise.  IWS  didn’t  seem  to  have  any  strategy  that  it  used  to  deliver 
audio  when  it  was  having  trouble  keeping  up.  The  transmissions  just  become 
impossible  to  understand. 
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9.  Remote  Room  Login 

Remote-room  login  tests  were  to  determine  how  much  time  it  would  take 
for  a  client  to  join  a  meeting  in  a  sidebar  room  on  the  server  which  was  not  its 
home  server.  Upon  joining  the  workspace,  all  data  was  pushed  to  the  new 
member  of  the  workspace  (time  varied  between  2-7  minutes  depending  on 
bandwidth).  If  more  clients  had  to  join  at  the  same  time,  then  the  process  for 
downloading  the  workspace  would  take  much  longer.  IWS  has  a  Quality  of 
Service  (QoS)  indicator  display  that  indicated  that  the  QoS  was  between  1-14 
when  room  login  was  attempted  by  the  remote  client. 

The  IWS  client  software  operated  on  the  client  computer  as  a  local 
application.  In  the  scenario  where  a  remote  client  logged  into  a  room  located  on 
the  local  server,  data  was  transferred  between  the  local  server  and  the  remote 
client.  When  delay  was  increased,  the  remote  client  could  not  join  the  local 
server  sidebar  room  1  for  five  minutes  at  256  kbps  bandwidth  and  2000ms  delay 
(4000ms  round  trip).  This  may  be  due  to  the  fact  that  previous  chat  conversations 
were  being  uploaded  to  sidebar  room  1.  There  is  no  mechanism  feature  for 
clearing  chat  from  a  room;  the  Placeware  server  must  be  stopped  and  re-started. 

Qther  operator  delays  with  respect  to  logging  into  a  remote  sidebar  room 
at  low  bandwidth  are  noted  as  follows.  At  9.6  kbps/10  one  observed  a  1min45s 
time  for  log  into  remote  server  sidebar  room.  At  4.8  kbps/10  one  observed  a  4 
minute  login  time.  Login  and  navigation  time  to  the  remote  server  sidebar  room  2 
was  4  and  a  half  minutes  at  9.6  kbps/1 000ms  to  log  in  as  opposed  to  1  minute  45 
when  the  delay  was  10ms. 

10.  Combined  Applications 

For  combined  application,  which  involves  testing  multiple  features 
simultaneously,  scenario  one  was  executed  for  a  duration  of  one  minute.  The 
features  combined  were  Chat  and  file  upload,  for  one  pair  of  clients  across  the 
WAN,  and  Whiteboard  and  Audio  for  the  other  pair  of  clients  across  the  WAN. 
Chat  input  consisted  of  one  operator  transmitting  a  string  of  chat  text  every  10 

seconds,  while  another  provided  voice  input,  and  another  added  an  image  in 
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JPEG  format  to  the  whiteboard.  For  the  file  upload  feature,  while  the  remote 
client  was  uploading  a  100K  file,  all  other  applications  were  virtually  stopped. 
After  the  upload  was  completed,  chat  whiteboard  and  audio  continued.  Thus,  it 
would  seem  that  the  file  upload  feature  has  priority  over  the  others.  A  summary 
of  the  combined  application  test  results  is  provided  in  Figure  13. 


□  128/10 
■  64/10 
□  9.6/10 


Figure  13:  Combined  Applications  Test  Results 

Results  for  the  9.6kbps/10  case  show  that  when  bandwidth  is  limited, 
some  applications  have  priority  over  other  ones,  because  during  the  file  upload 
other  applications  appeared  to  stop. 


11.  Shared  View 

The  Shared  View  test  scenario  consists  of  one  client  presenting  a 
PowerPoint  presentation  with  voice  audio  from  a  room  at  one  server,  while  one  to 
three  clients  login  both  at  another  server  and  the  presenter’s  server  to  view  and 
listen  to  the  presentation.  The  shared  view  exercise  began  with  a  1  mb 
PowerPoint  file,  a  bandwidth  setting  of  128  kbps,  and  a  delay  of  500  ms.  The 
duration  of  the  presentation  was  three  minutes.  Slides  were  advanced  about 
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every  20  seconds  and  narration  occurred  for  most  of  the  time  with  interruptions  of 
about  3  seconds  to  change  slides.  The  lower  end  of  the  bandwidth  delay  testing 
was  conducted  at  64  kbps  bandwidth  and  800  seconds  delay.  At  this  setting  it 
took  about  9-10  seconds  for  the  slide  to  advance  at  the  clients  viewing  the 
presentation.  With  as  few  as  two  clients  viewing  the  presentation  the  audio  was 
unsatisfactory.  Typically,  it  would  cut  out  while  the  slide  was  updating.  There 
was  virtually  no  delay  with  1  client  at  a  bandwidth  of  128  kbps,  but  a  4  second 
delay  when  two  clients  were  listening.  If  the  viewing  client  remotely  logged  into 
the  presenter's  server  this  appeared  a  little  less  efficient,  and  audio  deterioration 
was  observed  with  as  little  as  one  client  viewer.  At  this  point  when  deterioration 
was  observed,  the  communications  path  was  essentially  at  its  limit.  At  the 
128/500  bandwidth/delay  setting  3  clients  could  view  a  presentation,  and  voice 
quality  was  acceptable  with  one  or  two  breaks;  this  appeared  to  be  the  limit  for 
audience  members  at  this  bandwidth/delay  setting  before  significant  deterioration 
would  occur.  A  summary  of  the  combined  application  test  results  is  provided  in 
Figure14. 


□  b/w&delay  (kbps^ms) 
Bl28/500(1Client) 

□  128/500(2ciients) 

□  128/500(3ciients) 
B64/800(1cli6nt) 
I364/800(2clients) 


Figure  14:  Shared  Viewing  Test  Results 
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H.  ASSESSMENT  OF  IWS  PERFORMANCE 


Table  6  provides  an  evaluation  of  how  well  each  IWS  feature  supported 
the  technological  constraints  of  limited  bandwidth  and  network  congestion  or 
latency  experienced  under  real  world  conditions  are  supported.  Results  of  test 
run  to  discover  performance  thresholds,  including  robustness,  e.g.,  ability  to  view 
an  image  with  a  certain  amount  of  discrimination,  and  acceptability  of  audio  or 
video,  are  also  summarized  in  Table  6.  A  rating  of  pass  indicates  that  the  IWS 
feature  supported  the  performance  constraints 


Test  Name 

Test  Observations 

Rating 

Communications 

Resiliency 

4  seconds  to  re-federate  servers 

and  remote  client  connection 

rejected 

Fail 

Chat  Resiliency 

75  seconds  to  re-federate  servers 

and  remote  client  connection 

rejected 

Fail 

Remote  File  Transfer 

Resiliency 

4.5  seconds  to  re-federate 

servers  and  remote  client  was 

locked  up 

Fail 

Audio  Resiliency 

3  minutes  and  40  seconds  to  re¬ 
federate  servers  and  remote  client 

unable  to  join  sidebar  rooms 

Fail 

Whiteboard  Resiliency 

5  minutes  and  20  seconds  to  re¬ 
federate  servers  and  remote  client 

unable  to  join  sidebar  rooms. 

Objects  restored  when  remote 

client  log  out  and  log  in  performed 

Fail 

Chat 

No  problems  until  at  bandwidth  of 

Pass 
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1 .2  kbps  chat  began  to  see 

significant  delays  for  scenario  1 

and  when  the  bandwidth  was  9.6 

kbps  for  scenario  4 

File  Transfer  (sizes  100 

kb,  500  kb  and  1  mb) 

At  a  delay  of  10ms  (low)  the  data 

transfer  rate  approaches  the  limits 

of  the  communications  path. 

Pass 

Whiteboard 

At  bandwidth  of  128/10  ms,  when 

the  drawing  action  occurred  from 

the  remote  client,  there  was  a  3 

second  delay  before  rendering 

was  complete.  When  drawing  was 

initiated  by  the  local  client,  the 

rendering  was  instantaneous. 

Pass 

Video 

The  picture  appeared  to  update 

every  couple  of  seconds  even  at 

higher  bandwidth  of  2Mbps 

Application  had  to  be  stopped  and 

restarted. 

Fail 

Audio 

Choppy  and  slightly  delayed 

voice,  and  the  quality  seemed  to 

degrade  over  time.  At  32  kbps 

that  the  audio  quality  had 

deteriorated,  and  was  choppy  and 

reverberated.  At  9.6  kbps  the 

audio  was  not  functioning  at  all 

Fail 

Remote  Room  Login 

At  256  kbps  bandwidth  and 

2000ms  delay  (4000ms  round 

Fail 
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trip),  remote  client  could  not  join 
the  local  server  sidebar  room  1  for 
five  minutes.  Login  and 
navigation  time  to  the  remote 
server  sidebar  room  2  was  4  and 
a  half  minutes  at  9.6 
kbps/1 000ms  to  log  in  as  opposed 
to  1  minute  45  when  the  delay 
was  10ms. 


Table  6:  Summary  of  IWS  Performance  Test  Results 
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VI.  CONCLUSION 


In  conclusion,  collaborative  tools  can  facilitate  communication  among 
warfighters  distributed  throughout  the  world.  We  have  a  responsibility  to  ensure 
that  the  tools  are  reliable  and  not  to  be  a  hindrance  to  those  who  use  them;  that 
means  they  should  not  be  intrusive  in  daily  operations  and  support  mission 
requirements.  The  current  desire  to  use  software  systems  developed  for 
business  purposes  in  warfare  settings  may  encounter  obstacles.  It  is  imperative 
that  we  understand  the  promises  and  the  limitations  of  collaborative  tools  and 
their  impact  on  warfighter  effectiveness  in  a  heterogeneous  command  center 
environment.  This  thesis  has  presented  a  methodology  for  assessment  of 
performance  reliability  for  collaborative  tools. 

It  is  the  author’s  position  that  it  is  critical  that  a  standard  test  methodology 
be  used  when  evaluating  the  performance  of  a  collaborative  tool.  This  can 
provide  a  consistent  assessment  and  useful  insights  for  comparison  studies 
where  it  is  critical  that  the  correct  tool  be  selected  to  meet  stated  mission 
requirements.  This  may  also  have  implications  for  the  evaluation  of  collaborative 
tools  according  to  some  other  aspect  of  reliability  such  as  usability.  This  research 
shows  the  need  for  more  evaluation  of  collaborative  tools  used  in  military 
operations  and  shows  how  a  test  methodology  can  be  systematically  applied. 

There  is  clearly  a  need  for  more  research  to  be  conducted  on  performance 
of  collaborative  software  in  military  operations.  There  are  also  many  other 
performance  related  criteria  that  could  be  used  in  the  selection  of  a  collaborative 
tool.  This  thesis  has  explored  collaborative  performance  in  terms  of  bandwidth 
and  delay.  Future  work  many  include  application  of  the  test  methodology  to 
include  an  evaluation  based  on  usability,  ease  of  setup,  and  number  of  software 
inconsistencies  encountered.  Each  tool  would  receive  a  score  in  each  of  the 
areas  and  then  a  recommendation  could  be  made. 
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